In this work we propose a non-contrastive method for anomaly detection and segmentation in images, that benefits both from a modern machine learning approach and a more classic statistical detection theory. The method consists of three phases. First, features are extracted by making use of a multi-scale image Transformer architecture. Then, these features are fed into a U-shaped Normalizing Flow that lays the theoretical foundations for the last phase, which computes a pixel-level anomaly map, and performs a segmentation based on the a contrario framework. This multiple hypothesis testing strategy permits to derive a robust automatic detection threshold, which is key in many real-world applications, where an operational point is needed. The segmentation results are evaluated using the Intersection over Union (IoU) metric, and for assessing the generated anomaly maps we report the area under the Receiver Operating Characteristic curve (ROC-AUC) at both image and pixel level. For both metrics, the proposed approach produces state-of-the-art results, ranking first in most MvTec-AD categories, with a mean pixel-level ROC-AUC of 98.74%.

Code and trained models are available at https://github.com/mtailanian/uflow.

The detection of anomalies in images is a long-standing problem that has been studied for decades. In recent years this problem has received a growing interest from the computer vision community, motivated by applications in a wide range of fields, ranging from surveillance and security, to even health care. But one of the most common applications, and the one that we are especially interested in, consists in automatizing the quality control of products in an industrial environment. For this case, it is often very hard (or sometimes even infeasible) to collect and label a good amount of data with a representation of all kinds of anomalies, since by definition these are rare structures. For this reason, the major effort on anomaly detection methods has been focused on unsupervised and self-supervised non-contrastive learning, where only normal samples (i.e. anomaly-free images) are required for training.

Currently, the most common approach for anomaly detection consists in embedding the training images in some latent space, and then to evaluate how far a test image is from the manifold of normal images [cohen2020sub-spade, defard2021padim, tsai2022multi-mspba, yi2020patch-svdd, roth2022towards-patchcore, lee2022cfa].

Other approaches focus on learning the probability density function of the training set. Among these, three types of generative models became popular in anomaly detection in the last five years. On one hand, Generative Adversarial Networks [goodfellow2014generative, anogan] learn this probability in an implicit way, being able only to sample from it. On the other hand, Variational Auto-Encoders [kingma2013auto, yang2020dfr] are capable of explicitly estimating this probability density, but they can only learn an approximation, as they maximize the evidence lower bound. And finally, Normalizing Flows [dinh2014nice, dinh2016density-nvp, kingma2018glow] are able to explicitly learn the exact probability density function. Being able to do so has several advantages, as it allows to estimate the likelihood and score how probable it is that the tested data belongs to the same training distribution. A major advantage, that we exploit in this work, is the possibility to develop formal statistical tests to derive unsupervised detection thresholds. This is a key feature for most real-world problems, where a segmentation of the anomaly is needed. To this end, we propose a multiple hypothesis testing strategy based on the a contrario framework [desolneux2007gestalt].

In this work we propose U-Flow, a new self-supervised non-contrastive method that ensembles the features extracted by multi-scale image transformers into a U-shaped Normalizing Flow’s architecture, and applies the a contrario methodology for finding an automatic segmentation of anomalies in images. Our work achieves state-of-the-art results, widely outperforming previous works in terms of IoU for the segmentation, and exhibiting top performance in localization metrics in terms of ROC-AUC. The main contributions of this paper are:

We propose a novel feature extractor, based on image Transformers combined in a multi-scale configuration.

We leverage the well-known advantages of U-like architectures in the setting of Normalizing Flows (NF).

Thanks to a NF that captures information at multiple image scales, we are able to provide a multi-scale characterization of the space of anomaly-free training images, as a latent space where both intra and inter-scale features are statistically independent.

We propose a multiple hypothesis testing strategy that leads to an automatic detection threshold to segment anomalies, outperforming state-of-the-art methods.

In short, we propose an easy-to-train method, that does not require any parameter tuning or modification to the architecture (as opposed to other state-of-the-art methods), that is fast, accurate, and provides an unsupervised anomaly segmentation. Example results are shown in Figure 1.

The remainder of this paper is organized as follows. In Section 2 we discuss previous work related to the proposed approach. Details of the method are presented in Section 3. In Section 4 we present the results of our anomaly detection method, along with extensive comparisons to the state of the art. An ablation study, that analyzes the role of each component of the proposed architecture, is presented in Section 5. We conclude in Section 6.

The literature on anomaly detection is very extensive, and many approaches have been proposed. In this work we focus on non-contrastive self-supervised learning, where the key element is to learn only from anomaly-free images. Within this group, methods can be further divided into several categories.

Representation-based methods proceed by embedding samples into a latent space and measuring some distance between them. For example in CFA [lee2022cfa], the authors propose a method to build features using a patch descriptor, and building a scalable memory bank. A special loss function is designed to map features to a hyper-sphere and densely clusterize the representation of normal images. Also, the method has the possibility of adding a few abnormal samples during training, that are used to enlarge the distance to the normal samples in the latent space. Similarly, in [yi2020patch-svdd], the authors also construct a hierarchical encoding of image patches and encourage the network to map them to the center of a hyper-sphere in the feature space. This work later inspired [tsai2022multi-mspba], where the authors aim to learn more representative features from normal images, using three encoders with three classifiers. In SPADE [cohen2020sub-spade] the anomaly score is obtained by measuring the distance of a test sample to the nearest neighbors (NN) that were computed at the train stage over the anomaly-free images. Additionally, PaDiM [defard2021padim] proposes to model normality by fitting a multivariate Gaussian distribution after embedding image patches with a pre-trained CNN, and base the detection on the Mahalanobis distance to the normality. Then, PatchCore [roth2022towards-patchcore] further improves SPADE and PaDiM, by estimating anomalies with a larger nominal context and making it less reliant on image alignment.

Reconstruction-based methods ground anomaly detection in the analysis of the reconstruction error. By using an encoder-decoder architecture such as [yang2020dfr], or a teacher-student scheme like [yamada2022reconstructed-rstpm], and training the network with only anomaly-free images, the network is supposed to accurately reconstruct normal images, but fail to reconstruct anomalies, as it has never seen them. Learning from one-class data leaves some uncertainty on how the network will behave for anomalous data. To overcome this potential issue, some works manage to convert the problem into a self-supervised one, by creating some proxy task. For example in [li2021cutpaste], authors obtain a representation of the images by randomly cutting and pasting regions over anomaly-free images to synthetically generate anomalies.

Regarding the use of Normalizing Flows in anomaly detection, a few methods have been recently proposed [rudolph2021same-differnet, yu2021fastflow, gudovskiy2022cflow], among which CFlow [gudovskiy2022cflow] and FastFlow [yu2021fastflow] stand out for their impressive results. The former uses a one-dimensional NF and include the spatial information using a positional encoding as a conditional vector, while the latter directly uses a two-dimensional NF. In both cases, an anomaly score is then computed based on the likelihood estimation of test data. In this work, we further improve the performance of these methods, by proposing a multi-scale Transformer-based feature extractor, and by equipping the NFs with a fully invertible U-shaped architecture. These two ingredients allow us to embed these multi-scale features in a latent space where intra and inter-scale features are guaranteed to be independent and identically distributed Gaussian random variables. From these statistical properties, we can easily design a detection methodology that leads to statistically meaningful anomaly scores and unsupervised detection thresholds.

The proposed method is depicted in Figure 2, and consists of three main phases: 1) Feature extraction, 2) U-shaped Normalizing Flow, and 3) Anomaly score and segmentation. The three phases are presented in the following.

Due to the fact that anomalies can emerge in a variety of sizes and forms, it is essential to collect image information at multiple scales. To do this, the standard deep learning strategy is to use pre-trained CNNs, often a VGG [vgg] or any variant of the ResNet [resnet] architectures, to extract a rich image feature representation. More recently, with the development of image vision Transformers, some architectures, such as ViT [vit] and CaIT [touvron2021going-cait] are also being used, since the features generated from them better compress all multi-scale information into a single feature map volume.

In this work, we construct a multi-scale Transformer architecture to further enhance these representations. More specifically, we employ two CaITs at different scales, independently pre-trained on ImageNet [imagenet]. The input size of our method matches the highest resolution CaIT, and the images are down-sampled by two before feeding them into the lower resolution Transformer. The fact that both CaIT networks were trained independently could be somehow beneficial, in the sense that it might give the network more flexibility to attend to different structures in each Transformer.

We clearly show the benefits of the proposed multi-scale CaIT in the ablation study carried out in Section 5, where we show that the results obtained are superior than using each one of the Transformers alone, and far better than other ResNet variants. Although the architecture is presented agnostic to the feature extractor, it is important to remark that all results presented in Section 4 were obtained using the exact same architecture.

The rationale for using NFs in an anomaly detection setting is straightforward. The network is trained using only anomaly-free images, and in this way it learns to transform the distribution of normal images into a white Gaussian noise process. At test time, when the network is fed with an anomalous image, it is expected that it will not generate a sample with a high likelihood according to the white Gaussian noise model. Hence, a low likelihood indicates the presence of an anomaly.

This second phase is the only one that is trained. It takes as input the multi-scale representation generated by the feature extractor, and performs a sequence of invertible transformations on the data, using the NF framework.

State-of-the-art methods following this approach are centered on designing or trying out different existent multi-scale feature extractors. In this work, we further improve the approach by not only proposing a new feature extractor, but also a multi-level deep feature integration method, that aggregates the information from different scales using the well-known and widely used UNet-like [unet] architecture. The U-shape is composed of the feature extractor as encoder, and the NF as decoder. We show in the ablation study of Section 5.1 that the U-shape is a beneficial aggregation strategy and further improves the results.

The NF in this phase uses only invertible operations, and it is implemented as one unique graph, therefore obtaining a fully invertible network. Optimizing the whole flow all at once has a crucial implication: the Normalizing Flow generates an embedding that is independent not only inside each scale but also inter-scales, unlike CFlow [gudovskiy2022cflow] and FastFlow [yu2021fastflow]. It is worth mentioning that in all works so far, the anomaly score is computed first at each scale independently, using a likelihood-based estimation, and finally merged by averaging or summing the result for each scale. Because of the lack of independence between scales, these final operations, although achieving very good performance, lack of a formal probabilistic interpretation. The NF architecture proposed in this work overcomes this limitation; indeed, it produces statistically independent multi-scale features, for which the joint likelihood estimation becomes trivial.

Architecture. The U-shaped NF is compounded by a number of flow stages, each one corresponding to a different scale, whose size matches the extracted Transformer-based features (see Figure 2). For each scale starting from the bottom, i.e. the coarsest scale, the input is fed into its corresponding flow stage, which is essentially a sequential concatenation of flow steps. The output of this flow stage is then split in such a way that half of the channels are sent directly to the output of the whole graph, and the other half is up-sampled, to be concatenated with the input of the next scale, and proceed in the same fashion. The up-sampling is also performed in an invertible way, as proposed in [jacobsen2018revnet], where pixels in groups of four channels are reordered in such a way that the output volume has 4 times fewer channels, and double width and height.

Each flow step has a size according to its scale, and is compounded by Glow blocks [kingma2018glow]. Each step combines an Affine Coupling layer [dinh2014nice], a permutation using 1×1 convolutions, and a global affine transformation (ActNorm) [dinh2016density-nvp]. The Affine Coupling layer uses a small network with the following layers: a convolution, a ReLU activation, and one more convolution. Convolutions have the same amount of filters as the input channels, and kernels alternate between 1×1 and 3×3.

To sum up, the U-shaped NF produces L white Gaussian embeddings z1,…,zL, one for each scale l, zl∈RCl×Hl×Wl. Here we denote by (i,j) a pixel location in the input image, and by k the channel index in the latent tensor. Its elements zlijk ∼ N(0,1), 1≤i≤Hl, 1≤j≤Wl, 1≤k≤Cl, are mutually independent for any position, channel and scale i,j,k,l.

The last phase of the method is to be used at test time when computing the anomaly map and its corresponding segmentation. Thanks to the statistical independence of the features produced by the U-shaped NF, the joint likelihood of a test image under the anomaly-free image model is the product of the standard normal marginals.

More precisely, this third and last phase of our method produces two outputs (see Figure 2), described below.

The first output is an anomaly map that associates to each pixel (i,j) in the test image, the same likelihood-based Anomaly Score proposed in FastFlow and CFlow,

Note that this score does not correspond exactly to the anomaly-free likelihood, mainly since it averages the (unnormalized) likelihood of each scale, instead of computing their product. This of course lacks of a clear probabilistic interpretation, but produces high quality anomaly maps and allows us to compare our approach to other state-of-the-art methods.

In order to obtain a more formal and statistically meaningful result, the output of this third phase produces also a second anomaly score called Number of False Alarms (NFA), computed following the a contrario framework. Moreover, this framework permits to derive an unsupervised detection threshold on the NFA, that produces an anomaly segmentation mask.

The a contrario framework [desolneux2007gestalt] is a multiple hypothesis testing methodology that has been successfully applied to a wide variety of detection problems, such as alignments for line segment detection [von2008straight], image forgery detection [marina], and even anomaly detection [tailanian2022contrario-book], to name a few.

The a contrario framework is based on the non-accidentalness principle. Given the fact that we do not usually know how the anomalies look like, it focuses on modeling normality, by defining a null-hypothesis (H0), also called background model. Relevant structures are detected as large deviations from this model, by evaluating how likely it is that an observed structure or event E would happen under H0. One of its most useful characteristics is that it allows to automatically set a detection threshold, by controlling the NFA, defined as:

where NT is the number of events that are tested, and PH0(E) (also noted just as P in the following), is the probability of the event E to happen as a realization of the background model. Besides providing an automatic threshold, the NFA value itself has a clear statistical meaning: it is an estimate of the number of times, among the tests that are performed, that such tested event could be generated by the background model, namely under normality assumptions. A low NFA value means that the observed pattern is too unlikely to be generated by the background model, and therefore indicates a possible anomaly.

In our anomaly detection setting, as the embedding produced by the U-Flow are zlijk ∼ N(0,1),i.i.d., we characterize normality by defining a background model on (zlijk)2 to follow a Chi-Squared distribution of order 1,

Then, instead of basing the criterion only on the observed pixel, we consider a block Blij of size wl×wl×Cl centered on pixel (i,j) and spreading through all channels, as shown in Figure 3, and define a set of candidates SBlij as the voxels inside the block with suspiciously high values, according to our normality assumption given by (3):

with i′∈[i−wl/2,i+wl/2], j′∈[j−wl/2,j+wl/2], k∈[1,Cl], and τ fixed to represent a p-value of p=0.9 in the X2(1) distribution.

Under the normality assumption, all candidates are distributed uniformly. Therefore, we base our approach to detect anomalies on the concentration of these candidates. Given an observed number |SBlij| of such points within Blij, the probability that the background model produces at least this number of such points within Blij, which we denote by Plij, is given by the tail of the Binomial distribution: Plij=B(|SBlij|,w2lCl,1−p). However, as the amount of experiments is usually too high, and due to numerical instabilities, we instead evaluate its average by channel:

Note that we want to evaluate the Number of False Alarms, which is an upper bound on the number of false detections, and therefore the average value |SBlij|/Cl is a conservative estimation.

Finally, we merge the results of different scales using the independence resulting from the NF, as: Pij=∏Ll=1Plij. The number of tests we perform is NT=∑lHlWl. As a result, following (2) and (5), the logarithm of the NFA value for the pixel (i,j) is given by:

We evaluate our method on MvTec-AD [bergmann2019mvtec], the most widely used benchmark for anomaly detection, as most state-of-the-art anomaly detection methods report their performances on this dataset. MvTec-AD consists of 15 categories from textures and objects, simulating real-world industrial quality inspection scenarios, with a total of 5000 images. Each category has normal (i.e. anomaly-free) images for training, a testing set with different kinds of anomalies, and some more normal images.

For a fair comparison, we adopt the ROC-AUC metric (the area under the Receiver Operating Characteristic curve), as it is the most used metric in the literature. The ROC-AUC results for the pixel-level (i.e. anomaly localization) and image-level (i.e. anomaly detection) are presented in Table 1. For the pixel-level metric, U-Flow achieves state-of-the-art results, even outperforming all previous methods on average. Regarding the image-level results, we simply used as image score as the maximum value of the anomaly score at the pixel level, achieving state-of-the-art results.

In addition, besides obtaining excellent results for both pixel and image-level ROC-AUC, our method presents another significant advantage with respect to all others: it produces a fully unsupervised segmentation of the anomalies, and significantly outperforms its competitors in terms of IoU, as shown in the next section.

Giving an operation point is crucial in almost any industrial application. Most methods in the literature focus on the ROC-AUC metric and do not provide detection thresholds or anomaly segmentation masks. Actually, some methods do provide segmentation results, but these are not consistent with a realistic scenario: they are obtained using an oracle-like threshold that is computed by maximizing some metric over the test images [roth2022towards-patchcore, tsai2022multi-mspba, gudovskiy2022cflow].

In this section we report results on anomaly segmentation based on an unsupervised threshold obtained by setting NFA=1 (log(NFA)=0). As explained in Section 3.3.2, this threshold means that in theory, we authorize at most, on average, one pixel as false anomaly detection per image. Results for the IoU metric are shown in Table 2. We limit this comparison to the two top-performing methods. As the state-of-the-art methods to which we compare do not provide detection thresholds, we adopt two strategies: (i) we compute an oracle-like threshold which maximizes the IoU for the testing set, and (ii) we use a fair strategy that only uses training data to find the threshold. In the latter, the threshold is set to allow at most one false positive in each training image, as it would be analogous of setting NFA=1 false alarm. For completeness, we also included in the table (“Ours AS.” column) the results using this fair strategy over our anomaly score computed following (1). As can be seen, our automatic thresholding strategy significantly outperforms all others, even when comparing with their oracle-like threshold.

It is worth mentioning that, since the trained models for FastFlow and CFlow are not available, it was needed to re-train for all categories. To ensure a fair comparison, we performed numerous trainings until we obtained comparable ROC-AUC to the ones reported in the original papers. Note that for these methods, a different set of hyper-parameters is chosen for each category, varying sometimes even the number of flow steps and the image resolution, while for our method the architecture is always kept the same.

Finally, Figure 5 presents typical example result images from various categories, and their comparison with other methods, for visual evaluation.

The method was implemented in PyTorch [NEURIPS2019_9015-pytorch], using PyTorch Lightning [Falcon_PyTorch_Lightning_2019]. The NFs where implemented using the FrEIA Framework [freia], and for all tested feature extractors we used PyTorch Image Models [rw2019timm]. All trainings were performed in a GeForce RTX 2080 Ti.

For the multi-scale CaIT version, the input sizes are 448 and 224 pixels in width and height. The Normalizing Flow has 2 flow stages with 4 flow steps each. For computing the NFA, we used p=0.9 both for the Chi-Squared and Binomial distributions, and block sizes w1=5 and w2=3 for the fine and coarse scales, respectively.

Our method only use four flow steps in each scale. As a result it has less trainable parameters, in comparison with FastFlow and CFlow, as shown in Table 3.

In this section we study and provide unbiased assessments regarding some of the contributions of this work: the significance of the U-shaped architecture and the benefits of the multi-scale Transformer feature extractor. Both results are shown all together in Table 4, and explained in the following sections.

Also, to get some insights and visualize the distinction between normal and abnormal samples, we include Figure 4, which displays the distribution of the NFA-based anomaly score (−logNFA) for both normal and anomalous pixels. The solid vertical line represents the automatic threshold (logNFA=0), and a dashed line the oracle-like threshold, i.e. the best possible threshold for IoU on the test set. The distributions are clearly separated, and the automatic threshold is located very close to the optimal point, supporting the a contrario thresholding strategy.

One of the contributions of the presented work is to propose a multi-level integration mechanism, by introducing the well-known U-shape architecture design to the Normalizing Flow’s framework. In order to demonstrate that this architecture better integrates the information of the different scales, we compare the results obtained in terms of ROC-AUC against a modification of the architecture, in which each flow stage runs in parallel, and the per-scale anomaly maps are merged at the end by just averaging them, as done by other methods such as FastFlow [yu2021fastflow]. Additionally, we compare the results obtained using each scale separately. The results, presented in the top half of Table 4, show that the U-merging strategy improves the performance in almost all cases. Furthermore, this strategy where the output of one scale inputs the next one, allows to use less flow steps for each scale, resulting in a network with fewer parameters, as shown in Section 4.2.